Tenure AI reads every document the viewer can open - #307

"tenure ai should be activated everywhere not just the tenure AI chatbot",
with a screenshot of a document summary reading:

    Not available
    Summaries currently support text documents (this is application/pdf).
    PDF and Office support arrives with the document pipeline.

THE SECOND SENTENCE WAS NOT TRUE WHEN IT WAS WRITTEN. The pipeline is already
here: `_lib/content.ts` has parsed Word through mammoth, Excel through xlsx and
PowerPoint through JSZip since the viewer shipped. Three of the four formats
that refusal named could have been summarised that day.

What actually existed was a gate — `^(text\/|application\/(json|csv|xml))` —
written before the parsers and never revisited after them. The limit was in the
gate, not in the product, and a student was being told a roadmap to explain it.

── WHAT CHANGES ────────────────────────────────────────────────────────────

`documentTextFor(mime, bytes)` asks the parsers that already exist, so Word,
Excel, PowerPoint, CSV and text all summarise now. It adds no parsing of its
own — it asks the same parsers a different question. The viewer wants a docx as
HTML it can style; a reader wants the same docx as prose, so this uses
`extractRawText` rather than `convertToHtml`. Markup in a prompt is noise a
model spends attention discarding, and the sanitizer is not involved because
nothing here goes near `dangerouslySetInnerHTML`.

Sheet NAMES are kept. "Q3 budget" versus "Sheet1" is most of what a
spreadsheet's structure tells a reader, and a model that cannot see it is
reading a wall of numbers.

── PDF IS THE ONE THAT IS HONESTLY NOT HERE ────────────────────────────────

There is no PDF text extractor in this repo — `lib/export/pdf.ts` WRITES the
format and cannot read it, and the viewer shows a PDF by pointing a frame at
the bytes and letting the browser do the reading. Adding an extractor is a
dependency decision, so PDF refuses.

The refusal promises nothing this time. The old copy dated the fix to a
"document pipeline" a student has no way to ask about, and a promise a product
cannot keep is worse than a plain limit.

── EVERY REFUSAL SAYS WHICH ONE IT IS ──────────────────────────────────────

`documentTextFor` returns a REASON rather than a boolean, so the page can tell
apart things that used to collapse into one "Not available":

  · a PDF        — cannot read it yet; open it to read it in full
  · an image     — there is no text in a photograph to summarise
  · EMPTY        — a scan in a wrapper: nothing to summarise, which is not the
                   same as nothing we can open
  · too large    — over the parse ceiling `content.ts` already applies
  · unsupported  — a format, named in the reader's word for it, never as a
                   Content-Type

None of them prints `application/pdf` at a person. A reader shown a header
value has been handed an identifier and told it is an explanation.

── TRUNCATION IS REPORTED ──────────────────────────────────────────────────

A long document is cut at the read ceiling, and the card says so — "Generated
from the beginning of this document", with a line under the summary. A summary
drawn from part of a document and described as one drawn from the whole is
exactly the silent-coverage failure the export work has been fixing all day.

── VERIFICATION ────────────────────────────────────────────────────────────

13 tests, including a REAL .docx built in the test rather than committed as a
binary fixture — Word is the headline claim here and a test that only proves a
corrupt docx is refused does not support it. Mutating the classifier so every
format reads as text fails five of them.

Nothing throws: a corrupt file of a format we claim to read degrades to a card,
never a 500 on the page whose entire job is to be helpful.

tsc 0 errors · jest 417 suites / 6,664 tests green · next build exit 0 · eslint
clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The advisory gate failed #307, by name, and it was right.

`xlsx@0.18.5` carries two HIGH advisories that are REACHABLE and that npm
cannot fix — GHSA-4r6h-8v6p-xvw6 (prototype pollution) and GHSA-5pgg-2g8v-p4x9
(ReDoS). The registry's newest xlsx IS 0.18.5; the patched builds are published
only to cdn.sheetjs.com. So the repo ACCEPTS them rather than suppressing them,
and the acceptance is bounded by an enumerated list of parse sites: "it does
not assert the advisories are unreachable — they are — it asserts the blast
radius has not grown since the exposure was written down and accepted."

`documentTextFor` had added a second `XLSX.read`. Same bytes, same parser, same
trust boundary — and still a new entry on the list somebody has to re-argue
every time this is audited.

So the read moves into `_lib/content.ts`, which is already the accepted site and
whose own header says heavy parsers live there and nowhere a client bundle can
reach. `sheetsToText` is exported from there; `text.ts` no longer imports xlsx
at all. `sheet_to_csv` stays with the caller because it is not a parse entry
point — the advisories exempt workflows that do not read arbitrary files.

The blast radius is now exactly what it was: `advisory-recheck: OK - 4 xlsx
parse site(s), all within the accepted exposure.`

Worth saying plainly: adding the file to the allowlist would have passed the
gate too. It would also have been the wrong answer, because the gate is not
asking permission — it is asking whether the argument that justified accepting
a known-vulnerable dependency is still true. Widening the list makes the
argument weaker; not needing to widen it leaves it exactly as strong.

It also makes this module's own header honest. It claims to add no parsing and
to ask the existing parsers a different question, and for Word and PowerPoint
that was already true. For Excel it was not.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Two more review findings, and the first is the failure this whole change was
supposed to prevent.

── A SUMMARY OF A QUARTER OF A DOCUMENT, DESCRIBED AS THE WHOLE ────────────

`MAX_READ_CHARS` was 200,000. `summarizeDocument` slices to 24,000. So for
every document between those two numbers the extractor returned `truncated:
false`, and the card said "Generated from the document contents" over a summary
the model had written after reading the first eighth of it.

The truncation notice was in this change from the start, and its own commit
message says why: "a summary drawn from part of a document and described as one
drawn from the whole is exactly the silent-coverage failure the export work has
been fixing all day". It was doing that, in the code that says it must not — so
the notice was worse than absent, because it was evidence of care that was not
being taken.

`SUMMARY_INPUT_CHARS` is exported from `lib/ai.ts` and imported here. Two
numbers that must agree, in two files, is one number. The cut still happens in
the extractor, where the reason for it can be recorded.

── AND THE SPREADSHEET IS BOUNDED AS IT IS CONVERTED ───────────────────────

`sheet_to_csv` expands a worksheet's full `!ref` range, so a valid workbook
comfortably under `MAX_PARSE_BYTES` can produce a CSV very much larger — a
sparse sheet with one cell at ZZ100000 is tiny on disk and enormous as text.
Converting every sheet and slicing afterwards means the whole intermediate
string exists first, on the request thread.

`sheetsToText` now spends a budget sheet by sheet and stops. It deliberately
returns slightly OVER budget when the last sheet overshoots, because that
overshoot is what makes the caller's `truncated` flag true — a bound that
returned exactly the budget would be indistinguishable from a document that
happened to fit.

── THE TEST FOR THAT BOUND WAS VACUOUS, AND THE CONTROL CAUGHT IT ─────────

Worth recording. The first version went through `documentTextFor` and asserted
the result was `MAX_READ_CHARS` long — which is true whether or not the
conversion stopped early, because the caller slices at the end either way. It
passed with the bound removed.

It tests `sheetsToText` directly now, comparing twelve fat sheets against one
at the same budget, with the inverse case so it cannot pass by converting
nothing. Removing the break fails it.

The third finding on this PR — the xlsx parse site — was the one commit 1dd8cdd
already addressed by moving the read into the file that owns the accepted
exposure. `advisory-recheck: OK - 4 xlsx parse site(s), all within the accepted
exposure.`

tsc 0 · jest 417 suites / 6,669 tests green · eslint clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Two more findings, both real, and the second is the one that mattered.

── ONE WORKSHEET COULD STILL COST EVERYTHING ───────────────────────────────

The budget was checked BETWEEN sheets. `sheet_to_csv` walks the whole `!ref`
range in a single call before it returns, so one sparse worksheet declaring
A1:ZZ1000000 — tiny on disk, because almost every cell is absent — generated
and allocated all of it synchronously, and the loop only noticed afterwards.

A bound on the number of worksheets is not a bound on the work.

The range is clamped before conversion now: 1,000 rows and 64 columns from any
one sheet, whatever its `!ref` claims.

── THE DOCUMENTED OPTION FOR DOING THAT DOES NOT WORK ──────────────────────

`sheet_to_csv(ws, { range })` is documented and is silently ignored by this
build. Measured all three forms — a Range object, an encoded string, and a
numeric start row — and every one returned every row. Mutating `!ref` on the
worksheet before the call is what actually clamps it, so that is what this
does, with the measurement written down beside it so the next person does not
repeat it.

── A HEADING IS NOT CONTENT ────────────────────────────────────────────────

`--- Sheet1 ---` is a non-empty string, so a workbook of nothing but blank
sheets came back as readable text and was sent to a model to summarise a
heading. Blank sheets are skipped, and a workbook of only blank sheets now
reaches `reason: "empty"` — which the caller says differently from "cannot
open this", because they are different facts.

── THE CAP WAS SET BY A TEST TIMING, WHICH IS THE HONEST WAY TO SET IT ─────

First attempt: 5,000 x 256. That is 1.28 million cells, built to then be sliced
to the 24,000 characters a model actually reads, and the test proving the clamp
worked took SEVENTEEN SECONDS. A bound that is technically finite and
practically a stall is not a bound. 1,000 x 64 is several times what any reader
receives and returns in milliseconds.

The test itself was the other half of that seventeen seconds — the cost was
`XLSX.write` building a fixture with an absurd `!ref`, not the code under test.
An expensive test proving a bound against pathological input IS the pathological
input. Split into two cheap cases that prove the row clamp and the column clamp
separately: 16.8s -> 0.3s, and each clamp now has its own control.

All three changes have their own mutant, and each fails only its own test.

tsc 0 · jest 417 suites / 6,672 tests green · eslint clean · advisory-recheck OK.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>